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LONG-TERM  GOALS 

A  long-term  goal  of  this  project  is  to  develop  a  global  cloud-permitting  (-3 -km  resolution)  forecast 
capability  as  part  of  the  multi-agency  next-generation  Earth  System  Prediction  Capability  (ESPC) 
initiative.  Within  the  next  decade  or  less,  computational  platforms  with  more  than  1,000,000 
processors  will  be  commonly  available  and  high-fidelity  global  weather  forecasts  with  variable 
resolution  grids  at  the  mesoscale  or  cloud  permitting  scales  will  be  routinely  feasible.  Unfortunately, 
current  generation  weather  models  are  incapable  of  performing  such  forecasts  because  of  their  lack  of 
variable  resolution  grid  capabilities,  inadequate  numerical  accuracy,  and  computational  inefficiency.  A 
paradigm  shift  in  the  numerical  weather  and  climate  prediction  capabilities  of  the  US  Navy  and  the 
nation  is  needed  in  order  to  be  at  the  leading  edge  in  the  future.  The  time  is  right  and  critical  to  embark 
on  an  accelerated  development  path  of  a  modeling  system  with  next-generation  attributes  since  a 
decade  of  development  is  typically  required  to  advance  a  Numerical  Weather  and  Climate  Prediction 
(NWCP)  system  to  the  point  where  it  is  operationally  viable.  Global  and  regional  models  are  currently 
applied  as  separate  modeling  systems  at  many  operational  centers  such  as  the  Navy,  AFWA,  NCEP, 
with  the  global  models  providing  lateral  boundary  conditions  for  the  local  models  with  finer  spatial 
resolution  over  areas  of  interest.  Recently,  the  horizontal  spatial  resolution  of  global  models  used  in 
the  leading  centers  around  the  world  is  fast  approaching  the  grid  spacing  traditionally  used  by  the 
mesoscale  models,  which  is  on  the  order  of  10  km.  In  the  relatively  near  future,  when  the  resolution 
becomes  even  finer,  approaching  cloud  permitting  scales,  the  Nation’s  global  model  dynamical  cores 
will  need  to  account  for  non-hydrostatic  effects.  A  new  class  of  next-generation  models  is  emerging. 
The  long-term  goal  of  this  project  is  to  develop,  evaluate  and  eventually  implement  in  operations  a 
new  atmospheric  prediction  system  that  is  capable  of  scaling  efficiently  across  a  large  number  of 
processors  (up  to  1M  processor  counts  and  beyond),  and  has  the  capability  to  be  applied  on  the  globe 
at  high  resolution,  as  well  as  for  limited  area  applications  (e.g.,  urban  scale,  regional  climate  etc.). 

OBJECTIVE 

The  objective  is  to  develop,  evaluate  and  demonstrate  the  capabilities  of  a  new  generation  of 
atmospheric  dynamical  systems  that  allow  for  variable  resolution  on  the  sphere  or  for  limited  area 
domains,  are  highly  scalable,  and  eliminate  or  mitigate  spurious  problems  near  the  poles  of  the  globe. 
These  next-generation  models  are  non-hydrostatic  and  valid  for  predictions  ranging  from  nowcasting 
and  weather  time  scales  to  seasonal  and  decadal  climate  scales.  These  new  dynamical  systems  may 
comprise  the  atmospheric  component  of  the  multi-agency  Earth  System  Prediction  Capability. 
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APPROACH 


A  new  class  of  next-generation  weather  prediction  models  is  now  emerging.  These  models  include  the 
GFDL  High-Resolution  Atmosphere  Model  (HiRAM),  NCAR  Model  Prediction  Across  Scales 
(MPAS)  and  the  Navy’s  NEPTUNE  (Navy  Environmental  Prediction  sysTem  Utilizing  the  NUMA 
corE )  using  the  Naval  Postgraduate  School’s  Nonhydrostatic  Unified  Model  for  the  Atmosphere 
(NUMA)  dynamical  core.  These  models  are  nonhydrostatic,  which  is  needed  to  fully  address  the  Earth 
System  Prediction  Capability  (ESPC)  goals.  These  models  all  offer  selective  geodesic  grid  refinement, 
which  allows  for  variable  resolution  on  the  sphere  and  avoids  the  pole  problem. 

The  NEPTUNE  makes  use  of  the  NUMA  dynamical  core,  which  is  based  on  the  spectral  element 
method.  The  discretization  is  performed  on  hexahedral  elements  (rectangular  faces).  Since  the  spatial 
derivatives  are  computed  analytically,  there  is  no  need  for  spatial  staggering  of  prognostic  variables 
and  they  all  reside  on  the  same  nodal  points.  The  analytic  nature  of  derivatives  plays  an  important  role 
in  scalability  because  the  communication  stencil  among  neighbor  computational  cores  is  minimal.  The 
conservation  properties  are  accurate  to  the  machine  round  off.  The  dynamical  core  is  fully 
compressible  and  non-hydro  static.  The  physical  parameterizations  using  schemes  from  COAMPS, 

GFS,  and  NAVGEM  are  currently  being  implemented  and  tested  (moist  physics  has  already  been 
included).  The  model  simulations  can  be  performed  either  on  the  globe  while  utilizing  the  cubed- 
sphere,  or  a  limited  area.  Local  mesh  refinement  has  been  implemented  and  tested.  The  model  supports 
different  time  integration  schemes,  from  fully  explicit  (leapfrog,  Runge-Kutta),  semi-implicit,  with  the 
semi-implicit  correction  applied  in  all  three  dimensions  or  in  vertical  only,  to  split-explicit. 

Key  personnel:  James  D.  Doyle  (Naval  Research  Laboratory),  Sasa  Gabersek  (NRL),  Alex  Reinecke 
(NRL),  Kevin  Viner  (NRL),  Eric  Hendricks  (NRL/Naval  Postgraduate  School),  Frank  Giraldo  (Naval 
Postgraduate  School). 

WORK  COMPLETED 

A  major  part  of  our  effort  in  the  past  year  has  been  to  advance  the  Navy’s  next-generation  non¬ 
hydrostatic  modeling  system  NEPTUNE.  We  have  focused  on  building  the  infrastructure  to  run  real 
data  cases.  A  local  version  control  of  the  code  has  been  maintained  at  NRL,  which  is  accessible  by 
other  NEPTUNE/NUMA  users  including  NPS.  The  major  focus  of  work  was  getting  NEPTUNE  ready 
to  run  real  data  cases  on  massive  parallel  computers.  We  have  implemented  the  following  new  features 
and  improvements: 

•  A  new  time  integrator  based  on  the  Runge-Kutta  method  using  time-splitting,  where  the  fast 
modes  (related  to  sound  waves)  are  resolved  using  a  shorter  (small)  time  step;  the  scheme  is 
less  diffusive  and  it  allows  a  longer  (large)  time  step  which  dictates  when  the  physical 
parameterizations  are  computed. 

•  Input  and  output  communications  based  on  HDF5,  needed  for  massive  parallel  computing. 
Previously  all  the  data  was  gathered  on  a  single  compute  node  before  the  output,  which 
represented  an  efficiency  bottleneck.  With  HDF5,  the  I/O  operations  are  fully  parallel. 

•  New  domain  decomposition  based  on  p4est  needed  for  massive  parallel  computing. 

•  Digital  filter  to  provide  stable  initial  conditions  and  damping  of  fast  modes. 

•  Semi-analytic  metric  terms  to  ensure  stable  hydrostatically  balanced  initial  steady  state  on  the 
sphere. 
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•  A  stabilization  method  to  dampen  noise  at  short  wavelengths. 

•  A  switch  to  use  either  shallow-atmosphere  approximation,  or  full  deep-atmosphere  solution. 

We  have  continued  with  our  participation  in  the  NOAA  High-Impact  Weather  Prediction  Project 
(HIWPP)  to  test  and  evaluate  various  global  non-hydrostatic  dynamical  cores  under  development  at 
EMC,  GFDL,  ESRL,  NRL/NPS  and  NCAR  to  assess  their  potential  for  achieving  high-resolution 
global  weather  prediction  at  cloud  resolving  scales  (~3  km  horizontal  resolution).  The  following  tests 
have  been  carried  out  with  NEPTUNE/NUMA  in  the  past  year: 

•  An  idealized  baroclinic  wave  on  the  spherical  earth. 

•  An  orographic  gravity  wave  test  case  on  a  scaled  small  planet. 

•  An  idealized  supercell  test  case  on  a  scaled  small  planet. 

•  A  real  data  tropical  cyclone  test  case. 

These  test  cases  are  based  on  the  multi-agency  Dynamical  Core  Model  Intercomparison  Project 
(DCMIP)  (https://www.earthsystemcog.org/projects/dcmip/). 

RESULTS 

a.  Idealized  Test  Cases 


A  significant  improvement  in  the  baroclinic  wave  results  with  NEPTUNE  was  accomplished  by 
implementing  a  more  sophisticated  stabilization  method  (Fig  1). 


Longitude  (°)  Longitude  (°) 

Figure  1.  Surface  wind  speed  at  day  9  of  the  idealized  baroclinic  wave  test  case  using 
Nazarov  based  stabilization  (left)  and  Guba-based  stabilization  (right).  Horizontal 

resolution  is  approximately  120  km. 
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kinetic  energy 


A  more  quantitative  measure  of  improvement  is  supported  by  the  kinetic  energy  spectra  at  the  surface 
(Fig.  2).  With  the  initial  results  using  Nazarov-based  stabilization,  the  effective  resolution  of 
NEPTUNE  was  close  to  6Ax.  The  results  obtained  with  Guba  stabilization  indicate  the  effective 
resolution  at  4Ax. 
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Figure  2.  Surface  kinetic  energy  spectra  day  9  of  the  idealized  baroclinic  wave  test  case  using 
Nazarov  based  stabilization  (left)  and  Guba-based  stabilization  (right).  Average  horizontal  grid 

spacing  is  15  km. 

NEPTUNE  results  for  the  orographic  gravity  wave  test  and  idealized  supercell  test  (both  on  a  reduced 
radius  planet)  were  improved  compared  to  results  from  the  other  participating  models  when  the 
shallow-atmosphere  approximation  is  used  (the  only  option  available  for  the  rest  of  the  models).  When 
using  the  deep-atmosphere  equations,  the  standing  gravity  wave  train  has  a  constant  height-distance 
angle  and  is  rapidly  decreasing  in  amplitude.  With  the  shallow-atmosphere  equations,  the  wave  train 
becomes  more  horizontal  with  distance  from  the  orography  and  the  magnitude  of  the  perturbations  is 
less  evanescent  (Fig.  3).  The  shallow-atmosphere  switch  was  implemented  solely  for  the  model 
intercomparison  purposes. 
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Horizontal  distance  (km)  Horizontal  distance  (km) 

Figure  3.  Vertical  cross-section  of  vertical  velocity  (m/s)  for  the  Schaer  mountain  wave  case  when 
using  the  deep-atmosphere  (left)  and  shallow-atmosphere  approximation  (right). 


The  baroclinic  wave  test  was  also  used  for  scaling  purposes.  It  has  been  shown  that  the  NEPTUNE 
Spectral  Element  solver  (dynamics  only,  no  input/output)  has  nearly  ideal  scaling  (Fig.  4)  up  to  ~105 
computational  cores  (practical  upper  limit  of  the  computational  cores  on  the  testing  platform). 


Figure  4.  Neptune  scalability  (blue  line)  with  the  increasing  number  of  cores  compared  to  a  perfect 

simulation  rate  (black  line). 
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b.  NEPTUNE  Real-Data  Global  Simulation  Capability 

Hurricane  Sandy  was  chosen  for  the  real  data  capability.  Initial  conditions  were  based  on  GFS  fields, 
interpolated  onto  the  NEPTUNE  grid.  The  simulation  length  was  72  hours,  starting  on  October  24th  at 
18Z.  We  used  the  following  physical  parameterizations:  microphysics,  shallow  and  deep  convection, 
surface  fluxes,  surface  energy  model,  radiation,  subgrid-scale  mixing,  cloud  fraction,  and  planetary 
boundary  layer.  The  successful  simulation  represent  the  first  global  weather  prediction  obtained  with  a 
fully  three-dimensional  spectral  element  model,  using  a  whole  suite  of  physical  parameterizations. 


Figure  5.  Total  accumulated  precipitation  (mm)  18  hours  into  the  simulation ,  using  an  average 

horizontal  nodal  spacing  of  0. 75°. 


IMPACT/APPLICATIONS 

This  project  will  provide  a  new  generation  weather  and  climate  prediction  model,  which  will  accelerate 
the  US  Navy  capabilities  to  the  forefront.  A  new  generation  of  models  will  allow  us  to  simulate 
phenomena  across  both  spatial  and  temporal  scales.  On  fine  scales,  such  as  applications  over  coastal 
regions  or  complex  terrain,  the  off-line  forcing  from  the  lateral  boundary  conditions  will  no  longer  be 
required.  On  large  scales,  feedback  from  smaller  scales  will  be  handled  in  a  natural  and  explicit 
manner  instead  of  being  represented  through  crude  parameterizations. 

Information  superiority  is  a  key  element  of  the  DoD’s  strategic  plan  to  ensure  battlespace  dominance 
in  the  21st  century,  as  outlined  in  Joint  Vision  2020.  Situational  awareness  is  a  critical  aspect  of 
information  superiority.  A  high  fidelity  four-dimensional  (4D)  depiction  of  the  battlespace 
environment  using  a  variable  resolution  model  capturing  scales  ranging  from  planetary  to  clouds  can 
be  exploited  for  a  number  of  warfare  areas,  including  Intelligence,  Surveillance  and  Reconnaissance 
(ISR);  Strike  Warfare  (STW);  Anti-Surface  Warfare  (ASuW);  Anti-Submarine  Warfare  (ASW);  as 
well  as  to  prevent  damage  by  winds  and  seas  during  routine  Maritime  and  Aviation  Operations.  High- 
fidelity  forecasts  from  a  unified  modeling  system  will  provide  a  characterization  of  the  common 
operational  picture  as  well  as  support  tactical  decision  making  required  by  the  warfighter.  Great  strides 
have  been  made  towards  the  development  and  testing  of  a  new  generation  of  models  such  as 
NEPTUNE,  MPAS,  and  HiRAM,  but  much  of  this  technology  has  yet  to  be  validated  in  real-time 
environment  and  impact  operational  forecasts  within  the  Navy  or  any  other  operational  center.  These 
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emerging  technologies  can  be  exploited  over  the  next  several  years  to  enable  for  the  first  time  a  high- 
fidelity  global  depiction  of  the  battlespace  environment  to  serve  the  U.S.  Navy  and  DoD  needs. 

TRANSITIONS 

The  research  performed  under  this  project  will  be  transitioned  as  part  of  the  Earth  System  Prediction 
Capability  (ESPC)  program.  We  anticipate  the  new  atmospheric  model  will  be  transitioned  into 
operations  at  Fleet  Numerical  Meteorology  and  Oceanography  Center  (FNMOC)  in  the  next  7-10 
years.  Transitions  of  advanced  physical  parameterizations  will  be  made  to  the  NAVGEM  and 
COAMPS  6.4  programs  during  the  next  several  years.  The  ultimate  goals  are  to  develop  a  unified 
system  that  is  capable  of  accurate  coupled  predictions  on  the  globe  and  for  limited  area  high-resolution 
applications. 

RELATED  PROJECTS 

The  basic  NEPTUNE  development  is  advancing  under  various  programs  and  sponsorship. 
Development  in  6.2  is  taking  place  in  the  ONR  funded  program  “COAMPS-NG”  as  well  as  ONR 
sponsorship  at  the  Naval  Postgraduate  School.  The  MPAS  system  has  been  supported  by  DOE  and 
NSF,  while  the  HiRAM  system  has  primarily  been  supported  by  NOAA,  although  both  models  have 
leveraged  considerably  from  indirect  and  direct  sponsorship  outside  their  own  organizations.  We 
expect  close  coordination  with  the  ONR  DRIs  Physical  Parameterization  and  Seasonal  Prediction,  to 
draw  on  the  community  expertise  with  both  MPAS  and  HIRAM.  NRL  is  a  no-cost  collaborator  with  a 
number  of  proposals  for  the  ONR  Seasonal  Prediction  and  Physical  Parameterization  DRIs. 
Additionally,  this  work  will  benefit  from  ongoing  ONR  and  NRL  base  supported  6.2  COAMPS  and 
NAVGEM  efforts.  This  work  is  closely  coordinated  with  the  ongoing  6.4  Small  Scale  and  6.4  Global 
modeling  efforts.  In  particular,  we  are  leveraging  mature  physical  parameterization  research  that  is  in 
the  transition  process  in  these  projects  to  test  the  new  physics  in  NEPTUNE. 
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